Robust and Efficient Kernel Hyperparameter Paths with Guarantees

Algorithmically, many machine learning tasks boil down to solving parameterized optimization problems. The choice of the parameter values in these problems can have a significant influence on the statistical performance of the corresponding methods. Thus, algorithmic support for choosing good parameter values has received quite some attention recently, especially algorithms for computing the whole solution path of a parameterized optimization problem. These algorithms can be used, for instance, to track the solution of a regularized learning problem along the regularization parameter path, or for tracking the solution of kernelized problems along a kernel hyperparameter path. Since exact path following algorithms can be numerically unstable, robust and efficient approximate path tracking algorithms have gained in popularity for regularized learning problems. By now algorithms with optimal path complexity in terms of a guaranteed approximation error are known for many regularized learning problems. That is not the case for kernel hyperparameter path tracking algorithms, where the exact path tracking algorithms can also suffer from numerical problems. Here we address this problem by devising a robust and efficient path tracking algorithm that can also handle kernel hyperparameter paths. The algorithm has asymptotically optimal complexity. We use this algorithm to compute approximate kernel hyperparamter solution paths for support vector machines and robust kernel regression. Experimental results for these problems applied to various data sets confirm the theoretical complexity analysis. Proceedings of the 31 st International Conference on Machine Learning, Beijing, China, 2014. JMLR: W&CP volume 32. Copyright 2014 by the author(s).